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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  & 
Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 

•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters, 
Department  of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test,  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 
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The  GAO’s  11th  Annual  Assessment  of  Selected  Weapon 

Programs 


Michael  J.  Sullivan — Sullivan  is  the  director,  Acquisition  and  Sourcing  Management,  U.S. 
Government  Accountability  Office.  This  group  has  responsibility  for  examining  the  effectiveness  of  the 
DoD’s  acquisition  and  procurement  practices  in  meeting  its  mission  performance  objectives  and 
requirements.  In  addition  to  directing  reviews  of  major  weapon  system  acquisitions  such  as  the  Joint 
Strike  Fighter,  F-22,  Global  Hawk,  and  various  other  major  weapon  acquisition  programs,  Sullivan 
has  developed  and  directs  a  body  of  work  examining  how  the  DoD  can  apply  best  practices  to  the 
nation’s  largest  and  most  technically  advanced  weapon  systems  acquisition  system.  This  work  has 
spanned  a  broad  range  of  issues  critical  to  the  successful  delivery  of  systems,  including  technology 
development,  product  development,  transition  to  production,  software  development,  program 
management,  requirement-setting,  cost  estimating,  and  strategic  portfolio  management.  The  findings 
and  recommendations  from  this  work  have  played  a  major  role  in  the  department’s  recent  acquisition 
policy  revisions.  Most  recently,  he  has  directed  the  GAO’s  annual  assessment  of  major  weapon 
systems  programs  for  the  Congress  and  GAO’s  work  with  Congress  in  establishing  acquisition  policy 
reforms.  His  team  also  provides  the  Congress  with  early  warning  on  technical  and  management 
challenges  facing  these  investments.  Sullivan  has  been  with  the  GAO  for  24  years.  He  received  a 
bachelor's  degree  in  political  science  from  Indiana  University  and  a  master’s  degree  in  public 
administration  from  the  School  of  Public  and  Environmental  Affairs,  Indiana  University. 
[sullivanm@gao.gov] 

Abstract 

This  paper  reflects  the  GAO’s  observations  on  how  well  the  DoD  is  planning  and  executing  its 
$1 .602  trillion  portfolio  of  major  weapon  programs.  Although  the  total  projected  cost  of  the 
portfolio  remains  significant,  that  cost  has  declined  since  peaking  at  $1.75  trillion  in  2010  and 
is  currently  at  its  lowest  point  in  over  five  years.  In  addition,  the  number  of  programs  in  the 
portfolio  has  decreased  from  98  programs  in  2010  to  86  programs  in  2012.  DoD  weapon 
system  acquisition  represents  one  of  the  largest  areas  of  the  government’s  discretionary 
spending.  With  the  likelihood  of  decreased  defense  budgets  looming  in  the  near  future,  it  is 
imperative  that  the  DoD  continue  to  find  ways  to  reduce  cost  and  improve  efficiency. 

Introduction 

This  paper  reflects  the  GAO’s  observations  on  how  well  the  DoD  is  planning  and 
executing  its  $1,602  trillion  portfolio  of  major  weapon  programs.  Although  the  total  projected 
cost  of  the  portfolio  remains  significant,  that  cost  has  declined  since  peaking  at  $1.75  trillion 
in  2010  and  is  currently  at  its  lowest  point  in  over  five  years.  In  addition,  the  number  of 
programs  in  the  portfolio  has  decreased  from  98  programs  in  2010  to  86  programs  in  2012. 
DoD  weapon  system  acquisition — an  area  that  has  been  on  the  GAO’s  high-risk  list  for  more 
than  20  years — still  represents  one  of  the  largest  areas  of  the  government’s  discretionary 
spending.  Over  the  past  decade,  Congress  and  the  DoD  have  made  meaningful 
improvements  in  the  statutory  and  policy  frameworks  that  govern  weapon  system 
acquisitions  by  mandating  and  encouraging  a  more  knowledge-based  approach  to  the 
development  and  production  of  major  systems.  The  GAO  has  noted  in  the  past  that  practice 
has  lagged  behind  policy  in  certain  areas,  and  commensurate  improvements  in  program 
outcomes  have  not  been  evident.  However,  the  changes  in  the  DoD’s  portfolio  over  the  past 
few  years  indicate  that  some  improvements  are  being  realized.  With  the  likelihood  of 
decreased  defense  budgets  looming  in  the  near  future,  it  is  imperative  that  the  DoD  continue 
to  find  ways  to  reduce  cost  and  improve  efficiency. 

The  following  are  observations  on  (1)  the  cost  and  schedule  performance  of  the 
DoD’s  2012  portfolio  of  86  major  defense  acquisition  programs,  including  the  Missile 
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Defense  Agency’s  (MDA’s)  Ballistic  Missile  Defense  System  (BMDS);  (2)  the  knowledge 
attained  at  key  junctures  in  the  acquisition  process  for  40  weapon  programs  in  development 
or  early  production;  and  (3)  key  acquisition  reform  initiatives  and  program  concurrency.1  For 
a  more  detailed  discussion  of  each  of  the  observations,  see  GAO-1 3-294SP  (GAO,  2013). 

Data  from  three  sets  of  programs  provided  the  basis  for  the  observations: 

•  We  assessed  all  86  major  defense  acquisition  programs  in  the  DoD’s  2012 
portfolio  for  our  analysis  of  cost  and  schedule  performance.  To  develop  our 
observations,  we  obtained  cost,  schedule,  and  quantity  data  from  the  DoD’s 
December  2011  Selected  Acquisition  Reports  (SARs)  and  from  the  Defense 
Acquisition  Management  Information  Retrieval  Purview  system.  In  order  to 
fully  reflect  the  total  size  and  cost  of  the  DoD’s  portfolio,  we  included  the  cost 
of  BMDS — as  of  DoD’s  fiscal  year  2013  budget  submission — in  this  year’s 
assessment  of  the  changes  in  the  overall  cost  and  size  of  the  portfolio  over 
the  past  year.  However,  the  program  was  excluded  from  the  remainder  of  our 
analyses  because  no  acquisition  program  baseline  exists. 

•  We  assessed  40  MDAPs  that  were  mostly  between  the  start  of  development 
and  full-rate  production  for  our  analysis  of  knowledge  attained  at  key 
junctures  and  the  implementation  of  acquisition  reforms.  To  develop  our 
observations,  we  obtained  information  on  the  extent  to  which  the  programs 
follow  knowledge-based  practices  for  technology  maturity,  design  stability, 
and  production  maturity  using  a  data-collection  instrument.  We  also 
submitted  a  survey  to  program  offices  to  collect  information  on  systems 
engineering  reviews,  design  stability,  manufacturing  planning  and  execution, 
and  the  implementation  of  specific  acquisition  reforms.  We  received  survey 
responses  from  all  of  the  programs  from  August  to  November  201 2. 

•  In  addition,  we  assessed  17  future  major  defense  acquisition  programs  in 
order  to  gain  additional  insights  into  the  implementation  of  key  acquisition 
reform  initiatives.  To  develop  our  observations,  we  submitted  a  survey  to 
program  offices  to  collect  information  on  program  schedule  events,  costs,  and 
numerous  acquisition  reforms,  and  received  responses  from  all  17  future 
programs  from  August  to  October  2012. 

Observations  on  the  Performance  of  the  DoD’s  2012  Major  Defense  Acquisition 
Program  Portfolio 

The  overall  size  and  estimated  cost  of  the  DoD’s  portfolio  of  MDAPs  decreased  over 
the  past  year,  while  the  average  time  to  deliver  initial  capability  to  the  warfighter  increased 
by  one  month.  Our  analysis  of  the  DoD’s  2012  portfolio  allows  us  to  make  the  following  nine 
observations. 

1 .  The  DoD’s  2012  portfolio  of  major  defense  acquisition  programs  contains  86 
programs  with  a  combined  total  estimated  cost  of  $1 .602  trillion,  which  is  a 


1  Major  defense  acquisition  programs  are  those  identified  by  the  DoD  that  require  eventual  total 
research,  development,  test,  and  evaluation  (RDT&E)  expenditures,  including  all  planned  increments, 
of  more  than  $365  million,  or  procurement  expenditures,  including  all  planned  increments,  of  more 
than  $2.19  billion,  in  fiscal  year  2000  constant  dollars.  The  DoD  has  a  list  of  programs  designated  as 
pre-major  defense  acquisition  programs  (pre-MDAP).  These  programs  have  not  formally  been 
designated  as  MDAPs;  however,  the  DoD  plans  for  these  programs  to  enter  system  development,  or 
bypass  development  and  begin  production,  at  which  point  they  will  likely  be  designated  as  MDAPs. 
We  refer  to  these  programs  as  future  major  defense  acquisition  programs  throughout  this  report. 
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reduction  of  10  programs  and  more  than  $152  billion  from  201 1  levels.  This 
represents  the  smallest  portfolio  in  more  than  five  years.2 

2.  The  total  estimated  acquisition  cost  of  the  86  programs  in  the  201 2  portfolio 
decreased  by  $44  billion  over  the  past  year  while  the  delivery  of  initial 
operating  capability  slipped  by  one  month  on  average.3  When  assessed 
against  first  full  estimates,  the  total  cost  of  the  portfolio  has  increased  by  over 
$400  billion,  including  more  than  $90  billion  in  development  cost  growth  and 
nearly  $290  billion  in  procurement  cost  growth,  with  an  average  delay  of  27 
months  in  the  delivery  of  initial  operating  capability.4 

3.  Program  cancelations  and  restructurings  account  for  nearly  all  of  the  cost 
reduction  over  the  past  year. 

4.  Long-term  progress  of  the  Missile  Defense  Agency’s  $133  billion  BMDS 
cannot  be  assessed  because  insight  into  future  program  costs  is  limited  to 
the  five  years  covered  by  the  budget,  and  the  program  was  not  required  to 
establish  an  acquisition  program  baseline  when  it  began. 

5.  More  than  60%  of  the  programs  in  the  2012  portfolio  increased  buying  power 
over  the  past  year — as  measured  by  a  decrease  in  program  acquisition  unit 
cost — a  notable  improvement  when  compared  to  last  year,  when  more  than 
60%  of  the  programs  in  the  portfolio  lost  buying  power. 

6.  When  measured  against  cost  growth  targets  discussed  by  the  DoD,  the 
Office  of  Management  and  Budget  (OMB),  and  the  GAO,  the  portfolio’s 
performance  has  improved.  Only  15%  of  programs  exceeded  the  one-year 
target — down  from  40%  last  year — and  smaller  percentages  of  programs 
exceeded  targets  for  growth  both  in  the  past  five  years  and  since  first  full 
estimates. 

7.  Eight  of  the  10  costliest  programs  in  the  DoD’s  portfolio,  excluding  BMDS, 
reported  cost  reductions  over  the  past  year  totaling  nearly  $5  billion — about 
10%  of  the  portfolio’s  total  cost  reduction. 

8.  The  DoD  has  invested  more  than  $805  billion  in  its  2012  portfolio,  leaving 
over  $660  billion  remaining  to  be  funded,  excluding  BMDS.  More  than  90%  of 
the  remaining  funding  is  for  procurement,  with  more  than  60%  of  that  amount 
associated  with  the  10  costliest  programs  in  the  portfolio,  most  prominently 
the  Joint  Strike  Fighter. 

9.  Around  40%  of  the  funding  needed  to  complete  the  programs  in  the  portfolio 
represents  cost  growth  since  first  full  estimates. 

Observations  on  Knowledge  Attained  by  Programs  at  Key  Acquisition  Junctures 

Our  2013  assessment  continues  to  demonstrate  both  progress  and  a  significant 
need  for  programs  to  better  follow  a  knowledge-based  approach,  reducing  gaps  in 
technology,  design,  and  production  knowledge.  Knowledge  in  these  three  areas  builds  over 


2  All  dollar  figures  are  in  fiscal  year  2013  constant  dollars,  unless  otherwise  noted. 

3  In  addition  to  research  and  development  and  procurement  costs,  total  acquisition  cost  includes 
acquisition-related  operation  and  maintenance  and  system-specific  military  construction  costs. 

4  Our  discussion  of  cost  growth  since  first  full  estimates  does  not  include  the  BMDS,  as  the  program 
was  not  required  to  establish  an  acquisition  program  baseline  when  it  began  (see  GAO,  2012a,  for  an 
assessment  of  the  Missile  Defense  Agency’s  cost,  testing,  and  performance  progress  in  developing 
the  system). 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  360  - 


time — a  knowledge  deficit  early  in  a  program  can  cascade  through  design  and  production, 
leaving  decision-makers  with  less  knowledge  to  support  decisions  about  when  and  how  best 
to  move  into  subsequent  acquisition  phases  that  commit  more  budgetary  resources.  A 
knowledge-based  acquisition  approach  is  a  cumulative  process  in  which  certain  knowledge 
is  acquired  by  key  decision  points  before  proceeding.  Demonstrating  technology  maturity  is 
a  prerequisite  for  moving  forward  into  system  development,  during  which  the  focus  should 
be  on  design  and  integration.  A  stable  and  mature  design  is  likewise  a  prerequisite  for 
moving  forward  into  production  where  the  focus  should  be  on  efficient  manufacturing.  We 
assessed  the  knowledge  attained  at  key  junctures  in  the  acquisition  process  for  40  individual 
weapon  programs,  which  are  mostly  in  development  or  early  production.5  Not  all  programs 
included  in  our  assessment  of  knowledge-based  practices  provided  information  for  every 
knowledge  point  or  had  reached  all  of  the  knowledge  points — development  start,  design 
review,  and  production  start — at  the  time  of  our  review.  Our  analysis  allows  us  to  make  the 
following  three  observations: 

1 .  Many  of  the  programs  that  began  in  the  last  five  years  had  mature 
technologies  and  held  a  preliminary  design  review  prior  to  the  start  of 
development  (knowledge  point  1),  providing  a  better  foundation  to  avoid 
future  cost  and  schedule  problems. 

2.  Less  than  one  third  of  the  programs  that  provided  data  on  design  drawings 
released  actually  reported  having  a  stable  design  at  their  critical  design 
review  (knowledge  point  2),  and  the  use  of  other  knowledge-based  practices 
to  ensure  design  stability  at  this  critical  juncture  varied. 

3.  Many  of  the  programs  we  assessed  have  taken  or  plan  to  take  steps  to 
capture  critical  manufacturing  knowledge  prior  to  the  start  of  production 
(knowledge  point  3),  although  the  methods  used  varied. 

Observations  on  Implementation  of  Acquisition  Reform  Initiatives  and  Program 
Concurrency 

Over  the  past  several  years,  Congress  and  the  DoD  have  instituted  multiple 
initiatives  aimed  at  improving  the  way  the  department  does  business  by  driving  down 
acquisition  costs  and  ensuring  that  programs  are  more  affordable:  specifically  the  Weapon 
Systems  Acquisition  Reform  Act  of  2009,  the  reissuance  of  DoD  Instruction  5000.02 
(OUSD[AT&Lj,  2008),  and  multiple  “Better  Buying  Power”  memorandums  issued  by  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (OUSD[AT&L], 

2010a,  2010b,  2010c,  2012).  We  analyzed  survey  data  collected  from  40  current  major 
defense  acquisition  programs — the  same  programs  reflected  in  our  knowledge  point 
analysis — and  17  programs  identified  by  the  DoD  as  future  major  defense  acquisition 
programs,  regarding  the  implementation  of  key  aspects  of  these  reform  initiatives.  We 
focused  our  analysis  on  the  aspects  of  the  DoD’s  “Better  Buying  Power”  initiatives  and  the 
Weapon  Systems  Acquisition  Reform  Act  of  2009  aimed  at  ensuring  program  and  portfolio 
affordability,  controlling  cost  growth,  and  promoting  competition  throughout  the  acquisition 
life-cycle.6  In  addition,  we  assessed  the  amount  of  concurrency  between  developmental 


5  Because  knowledge  points  and  best  practices  differ  for  shipbuilding  programs,  we  exclude  the  six 
shipbuilding  programs  from  some  of  our  analysis  related  to  knowledge  point  2  and  knowledge  point  3. 

6  In  December  2012,  we  reported  on  the  DoD’s  implementation  of  the  Weapon  System  Acquisition 
Reform  Act  of  2009  and  noted  that  the  DoD  had  taken  steps  to  implement  fundamental  provisions  of 
the  Act,  and  that  the  DoD  was  taking  additional  steps  to  further  strengthen  its  policies  and  acquisition 
capabilities.  We  also  reported,  however,  that  the  DoD  still  faced  organizational,  policy,  and  cultural 
challenges  to  implementing  acquisition  reform  (GAO,  2012b). 
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testing  and  production  for  those  current  programs  beyond  knowledge  point  3. 7  We  have 
consistently  emphasized  the  importance  of  completing  developmental  testing  before 
entering  production  and  have  pointed  out  the  increased  risks  associated  with  concurrent 
testing  and  production.  Our  analysis  allows  us  to  make  the  following  five  observations: 

1 .  The  implementation  of  several  key  initiatives  in  the  Weapon  System 
Acquisition  Reform  Act  of  2009  aimed  at  increasing  program  knowledge  at 
development  start  varied  among  the  future  major  defense  acquisition 
programs  we  surveyed. 

2.  Around  half  of  the  current  and  future  programs  we  assessed  have 
established  affordability  requirements,  and  many  are  meeting  those 
requirements. 

3.  Almost  90%  of  the  current  MDAPs  we  assessed  have  conducted  “should 
cost”  analysis,  and  most  of  those  programs  noted  that  they  had  realized  or 
expected  to  realize  cost  savings  as  a  result. 

4.  Although  the  DoD  recognizes  the  need  for  and  benefits  of  competition  in 
weapon  system  acquisitions,  and  the  Weapon  Systems  Acquisition  Reform 
Act  of  2009  requires  programs  to  have  competitive  acquisition  strategies, 
many  of  the  programs  we  assessed  did  not  have  such  strategies  in  place. 

5.  Nearly  80%  of  the  programs  we  surveyed  that  were  in  production  reported 
that  30%  or  more  of  their  developmental  testing  had  been  or  was  going  to  be 
done  during  production,  despite  the  increased  risk  that  design  changes  and 
costly  retrofits  will  need  to  be  made. 
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